System and method for actionizing comments

ABSTRACT

A system and method for processing and actionizing structured and unstructured experience data is disclosed herein. In some embodiments, a system may include a natural language processing (NLP) engine configured to transform a data set into a plurality of concepts within a plurality of distinct contexts, and a data mining engine configured to process the relationships of the concepts and to identify associations and correlations in the data set. In some embodiments, the method may include the steps of receiving a data set, scanning the data set with an NLP engine to identify a plurality of concepts within a plurality of distinct contexts, and identifying patterns in the relationships between the plurality of concepts.

PRIORITY CLAIM

This application is a continuation-in-part of U.S. application Ser. No.16/156,336, filed Oct. 10, 2018, which is a continuation of U.S.application Ser. No. 15/277,892, filed Sep. 27, 2016, which claimspriority under 35 U.S.C. § 119(e) to U.S. Provisional Application Ser.No. 62/233,657, filed Sep. 28, 2015, each of which is expresslyincorporated by reference herein.

BACKGROUND

The present disclosure generally relates a system and method forprocessing and actionizing structured and unstructured experience data.The system and method described herein may be used for processingdisparate experience data sources such as user records, surveys, reviewsites, and social media.

SUMMARY

The present disclosure is a system and method for processing andactionizing structured and unstructured experience data. In someembodiments, a system may include a natural language processing (“NLP”)engine configured to transform a data set into a plurality of conceptswithin a plurality of distinct contexts, and a data mining engineconfigured to process the relationships of the concepts and to identifyassociations and correlations in the data set. In some embodiments, themethod may include the steps of receiving a data set, scanning the dataset with an NLP engine to identify a plurality of concepts within aplurality of distinct contexts, and identifying patterns in therelationships between the plurality of concepts.

One aspect of the present disclosure is a system for processing andactionizing experience data. The system comprises a server comprising anatural language processing (NLP) engine, and a relational database.Communications are received at the server, and each of thecommunications comprises comment data. The comment data from each of thecommunications is stored at the relational database. The comment datafrom each of the communications is parsed for individual phrases togenerate a plurality of phrases. One or more phrases are selected fromthe plurality of phrases based on a predetermined parameter. At the NLPengine one or more annotations are predicted for the one or more phrasesbased upon a score.

Another aspect of the present disclosure is a method for processing andactionizing patient experience data. The method includes receiving at aserver a plurality of communications. Each of the plurality ofcommunications comprises comment data. The method also includes storingthe comment data of each of the plurality of communication at arelational database. The method also includes parsing the comment datafor individual phrases to generate a plurality of phrases. The methodalso includes selecting one or more phrases from the plurality ofphrases based on a predetermined relationship. The method also includespredicting at a NLP engine one or more annotations for the one or morephrases based upon a score.

Additional features of the present disclosure will become apparent tothose skilled in the art upon consideration of illustrative embodimentsexemplifying the best mode of carrying out the disclosure as presentlyperceived.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The detailed description particularly refers to the accompanying figuresin which:

FIG. 1 is a diagrammatic flow chart of the overview of a system forprocessing and actionizing experience data;

FIG. 2 is an illustration of how an ETL job of FIG. 1 is completed usingsample input data and sample output data according to the embodiment ofthe present invention;

FIG. 3 is a flow chart of the hybrid NLP pipeline of FIG. 1;

FIG. 4 is an illustration of a response table into which data that hasundergone ETL is saved;

FIG. 5 is an illustration of a manual batching report table within whichthe preliminary analysis of pipelined data is represented;

FIG. 6 is an illustration of an annotation table which represents thefinal analysis of pipelined data;

FIG. 7 is an illustration for a web page for a dashboard of a system forprocessing and actionizing experience data;

FIG. 8 is an illustration for a web page for a dashboard of a system forprocessing and actionizing experience data; and

FIG. 9 is an illustration for a web page for a dashboard of a system forprocessing and actionizing experience data.

DETAILED DESCRIPTION

This present disclosure relates to a system and method for processingand actionizing structured and unstructured experience data. The presentdisclosure includes a hybrid NLP pipeline, which, combined with machinelearning and crowd sourcing, recognizes the sentiments, themes, andnamed entities within the data. Pipelined data is then visualized on auser dashboard, outlining areas where the user has performed well andareas where the user can improve.

In the present disclosure, the terms “field,” “data element,” and“attribute” may be used as synonyms, referring to individual elements ofdigital data. Aggregates of data elements may be referred to as“records” or “data structures.” Aggregates of records may be referred toas “tables.” Aggregates of tables may be referred to as “databases.”“Data mining” is, for example, an analytic technique to dynamicallydiscover patterns in historical data records and to apply propertiesassociated with these records to production data records that exhibitsimilar patterns. “Unordered,” for example, means that the chronology ofa record's attributes is not known. “Unstructured,” for example, meansthat the attributes of a comment or phrase are not known.

FIGS. 1-3 show an embodiment of the present disclosure that isvisualized, for example, on the dashboard 130. FIG. 1 provides anoverview of the system. It illustrates how input, that is, comments,from various sources are aggregated to then undergo ETL 104 to becomestructured and ordered information. In one example, the sources may behospital surveys, verbatims, and social media posts. In another example,the sources may be employment surveys, exit surveys, and social mediaposts. In yet another example, the sources may be airline, hotel,restaurant, or financial services customer surveys or online reviews.This information next enters the hybrid NLP pipeline 112, where commentsare segmented into phrases, the phrases studied to recognize sentiment,theme, and any named entity, which are finally approved and reflected ina batching report 128. The finalized information within the batchingreport is then visualized on a user's dashboard 130 or provided via anAPI, providing business intelligence for the user to act upon theinputted patient feedback.

FIG. 2 illustrates the ETL process of FIG. 1. Box 101 represents patientfeedback data in HTML form, such as posts from social media. This datais unstructured and unordered. In one example, box 102 represents datafrom hospital surveys. This data is structured, but unordered. Inanother example, box 103 represents data from adaptive rounding surveys.This data is structured and ordered; thus requiring minimal extractionand transformation.

During extraction, 105 through 107, information from the data sources isconverted from unstructured form to structured form. Duringtransformation, 108 to 110, it is also converted from unordered toordered form. Once input is extracted and transformed, loading 111occurs wherein its attributes are recognized and loaded into theresponse table, shown in FIG. 4, thereby establishing the input'srecord. As shown in FIG. 4, each row of the table represents a differentrecord. Each record may comprise at least one of an input's text anddate (i.e., its timestamp). Depending on the data source, however, therecord may also contain an input's unit, the name of a professionalabout whom the input was written, and the input author's demographics.

Next, as shown in FIG. 3, from the response table of FIG. 4, a subset ofcomments, or a comment batch 113, enters the hybrid NLP pipeline. Thehybrid NLP pipeline operates on a server. Comments are parsed forindividual phrases based upon punctuation 114. A phrase includes eithera full sentence or a part of a sentence that represents a completethought. From this, a phrase batch is developed 115 for machine learninganalysis. Each phrase is given a phrase identification number. Throughmachine learning, a phrase's sentiment, theme, and any named entitytherein is predicted based upon a percentage of likelihood, 116 through118. The machine learning for these areas, 116 through 118, ispreferably done simultaneously.

Machine learning may produce a prediction for anyone of the areas onlyif the percentage of likelihood satisfies a predetermined relationship,for example a threshold percentage. This prediction along with thepercentage of likelihood on which the prediction is based will be notedin the machine learning reports for sentiment 119, theme 121, and namedentities 123. If the predetermined relationship is not satisfied,however, machine learning will not produce a prediction for an area.Instead, the phrase may be sent to be crowd sourced or individuallyreviewed to determine what the ambiguous sentiment, theme, or namedentity is, 120, 122, or 124. Then, upon the majority vote ofcrowd-sourcers or individual review, a prediction will be made andreflected in the respective crowdsourcing report, 125 through 127. Othercriteria than majority, such as specific percentages, may be usedinstead of simple majority. Alternatively, instead of sending the phrasefor crowd-sourced or individual review, a rule or set of rules may beused to determine what the ambiguous sentiment, theme, or named entityis 120, 122, or 124.

For example, in regards to sentiment, machine learning attempts topredict whether a phrase is positive or negative 116. However, if thethreshold percentage is set at 90% and if machine learning can onlypredict that the likelihood that a phrase is positive is 84%, it willnot produce a sentiment prediction 119. The phrase will instead be crowdsourced 120 and voted upon to establish its sentiment prediction, whichwill be reflected in the sentiment crowd report 125. The predictionresults of the machine learning reports and the crowd sourcing reportsfor each phrase are next compiled and reviewed during the batching step128 (manual or automatic). Preferably, for manual batching, there is ahuman review of the predictions in which any area's-sentiment's,theme's, or named entity's machine learning or crowd sourcing predictionis marked as true or false, or, in other words, approved or rejectedautonomously and with final authority at step 128. The purpose of thestep is to ensure accuracy. Upon completion, as shown in Table Two ofFIG. 5, each phrase is then listed by its identification number and thephrase's sentiment and any named entity therein is reflected. The phraseis also given a primary and possibly a secondary tag. A primary tagreflects what a phrase is mainly about. This usually corresponds withthe subject of a phrase. A secondary tag is optional and reflects aphrase's general theme. A phrase can have multiple primary or secondarytags.

Once the results of the batching (manual or automatic) are finalized,they are inputted into the annotation table 129 (and as shown in TableThree of FIG. 6). At the annotation table 129, each phrase from thecomment batch 113 is listed by phrase identification number and mirrorsthe records of the manual batching report of FIG. 5, except that thereis no approval step. The information reflected in the annotation tableof FIG. 6, is thus the finalized records of each phrase, which is thenvisualized on a user's dashboard 130 to provide the user with businessintelligence and allow the user to actionize patient feedback, 101through 103.

The NLP engine preferably predicts if a phrase is positive or negative.The NLP engine preferably tags each phrase based on a subject matter ofthe phrase.

The system further comprises a dashboard for providing businessintelligence for the user to act upon the inputted patient feedback.This information can also be provided via an API.

The comment data of each of the plurality of communications ispreferably extracted at server, wherein the extracted texts aretransformed to a format compatible with a target, and wherein thetransformed texts are loaded into a plurality of tables of therelational database.

The comment data from each of the plurality of communications ispreferably parsed for individual phrases based upon punctuation orlinguistic structure. A phrase's annotation preferably comprises one ofa phrase's sentiment, theme, or any named entity therein. The pluralityof communications is preferably collected from publicly available dataor uploaded from an entity. In one example, the entity may be ahospital, medical provider, employer, third party service provider ordata collection service or the like. Phrases that receive a completeannotation (sentiment, primary tag, secondary tag with an optionaldriver), also generate an improvement action because the logic of theannotation is structured such that there is one improvement for eachunique annotation. Each phrase of the plurality of phrases is preferablyprovided with an identification.

A system for processing and actionizing experience data comprises aserver comprising a natural language processing (NLP) engine, and arelational database. The server operates, for example, on AMAZON webservices (“AWS”), and the engine is written in PYTHON running on theserver, for example. The relational database is, for example, Postgresq1running on AWS. The communications received at the server are preferablyscraped comments from Internet web sites such as YELP, ZOCDOC, or thelike. Each of the communications comprises comment data such as surveycomments from patient satisfaction surveys or employment surveys orcustomer feedback surveys. The relational database preferably mapssources to their comments. The comment data from each of thecommunications is parsed for individual phrases to generate a pluralityof phrases, preferably by linguistic structure. One or more phrases areselected from the plurality of phrases based on a predeterminedparameter. At the NLP engine one or more annotations are predicted forthe one or more phrases based upon a predetermined relationship, forexample a score, such as a sentiment and a theme assigned to each phrasewith a score meeting a criteria, such as being greater than 0.5. Anyother suitable relationship may be used.

For example, a web-scraped comment from YELP from a patient that stated“My nurse was mean and she hit me,” is received at the server. A surveycomment that, “I like the food. I do not like the beds” is also receivedat the server. The comments are parsed into: My nurse was mean; she hitme; I like the food; and I do not like the beds. In this example,phrases are selected based on linguistic characteristics—the contenthaving more than three words. So the following phrases are selected: Mynurse was mean; I like the food; and I do not like the beds. Next, atthe NLP engine one or more annotations are predicted for the one or morephrases based upon a score. Negative (0.7)+Attitude (0.6):Negative+Attitude: My nurse was mean. Positive (0.8)+Food (0.4):Positive: I like the food. Negative (0.3)+Comfort (0.9): Comfort: I donot like the beds.

The hybrid natural language processing pipeline (“pipeline”) is anatural language batch-processing system. The input for the pipeline isorganized as “batches” (groups) of “documents.” Alternatively, thepipeline processes one batch at a time. Pieces of each document arereferred to as “phrases.” A phrase can be any text within the document,including the entire document. The pipeline's primary function is totake batches of comments, split the comments into phrases and thenassign an annotation to each phrase. An annotation, for example, is acomplete set of categorical or numeric labels.

The hybrid natural language processing pipeline combines four componentsto generate high quality annotations: Rulebased, Machine, Crowd, andAnnotation. The rulebased component uses rules to generate individuallabels for any annotation type with 100% certainty. The machinecomponent generates labels for any phrase with variable certainty. Thecrowd component generates labels for any phrase using an open-call poolof workers. The annotation component uses logic to decide which labelsfrom which components are ultimately assigned to each phrase.

A method for processing and actionizing experience data begins withcollecting data from multiple sources: Internal sources, such asgrievances, nurse rounding, call transcripts, etc; Public sources, suchas Hospital Compare, CMS, Doctor Review Sites and Social Media; andSurveys, such as CAHPS, HCAHPS, CG-CAHPS and custom surveys, as well asemployee engagement surveys and customer feedback surveys. The next stepis to Perform Annotations with Hybrid NLP Pipeline. The annotationspreferably comprise: Themes, Named Entities, Sentiment, CategoryDiscovery, and Category Annotation. The next step is to generateimprovement suggestions for each category. The next step is to combinequalitative and quantitative data with new data collected by repeatingthe process from the first step.

FIG. 7 is an illustration for a web page 700 for a dashboard of a systemfor processing and actionizing experience data.

FIG. 8 is an illustration for a web page 800 for a dashboard of a systemfor processing and actionizing experience data.

FIG. 9 is an illustration for a web page 900 for a dashboard of a systemfor processing and actionizing experience data.

The server includes a CPU component, a graphics component, PCI/PCIExpress, RAM memory, non-removable storage, removable storage, NetworkInterface, including one or more connections to a fixed network, and aSQL database. Included in the memory, are the operating system, the SQLserver, and computer programs. The data server also includes at leastone computer program configured to receive data uploads and store thedata uploads in the SQL database. The SQL server comprises of othercomponents of SQL server that can be installed separately from the SQLdatabase engine.

Each of the interface descriptions preferably discloses use of at leastone communication protocol to establish handshaking or bi-directionalcommunications. These protocols preferably include but are not limitedto XML, HTTP, TCP/IP, Serial, UDP, FTP, Web Services, WAP, SMTP, SMPP,DTS, Stored Procedures, Import/Export, Global Positioning Triangulation,IM, SMS, MMS, GPRS, and Flash. The databases used with the systempreferably include but are not limited to MSSQL, Access, MySQL,Progress, Oracle, DB2, Open Source DBs and others. Operating system usedwith the system preferably include Microsoft 2010, XP, Vista, 2000Server, 2003 Server, 2008 Server, Windows Mobile, Linux, Android, Unix,I series, AS 400, and Apple OS.

The underlying protocol at the server, is preferably Internet ProtocolSuite (Transfer Control Protocol/Internet Protocol (“TCP/IP”)), and thetransmission protocol to receive a file is preferably a file transferprotocol (“FTP”), Hypertext Transfer Protocol (“HTTP”), Secure HypertextTransfer Protocol (“HTTPS”) or other similar protocols. The transmissionprotocol ranges from SIP to MGCP to FTP and beyond. The protocol at theserver is preferably HTTPS.

Natural language processing (“NLP”) is a field of computer science,artificial intelligence, and linguistics concerned with the interactionsbetween computers and human languages. It involves the processing of anatural language input. A natural language input is generally languageused by a person (as opposed to a computer language or other artificiallanguage), including all of the idioms, assumptions and implications ofan utterance in a natural language input. Natural language processingimplemented by a computer is typically an attempt to determine themeaning of a natural language input such that the natural language inputcan be “understood” and/or acted on by the computer. To interact withhumans, natural-language computing systems may use a data store that isparsed and annotated.

In the healthcare and employer industries, for example, there is a needfor systems and methods that are able to rapidly parse, combine, andinterpret multiple structured and unstructured data sources. Healthcareinformation, such as information related to a patient's care experienceand satisfaction, is fractured across many isolated data stores invarying formats. The same may be true regarding employment or employeesatisfaction. To compound the problem, even when data is available,there are no easily available means of processing this data with a highdegree of accuracy or efficiency.

Moreover, in health care data management systems today, only about 20%of data is structured or machine-readable. Information that is notstructured or machine readable may be ignored or unusable inconventional analytics systems. Online data sources, such as doctorreview sites and social media, consist of largely unstructured data.Additionally, data collected from surveys or other public and privatesources is often a mixture of both unstructured and structured data thatvaries between data stores. Due to lack of interoperability betweenthese data stores and formats, these sources have not been analyzed inconjunction with one another.

Significantly, online data sources have risen in importance forhealthcare providers, similar to most customer-focused industries.Online data sources may also be important for employers and otherindustries that collect customer feedback. Data from online sources isextracted, transformed, and loaded into a structured/compatible form.Extract, Transform, Load (ETL) jobs extract data from a source,transform the extracted data using one or more transformations to aformat compatible with a target, and load the data into the target, forexample a target database. Extraction refers to actually obtaining thedata from individual data sources. Transformation indicates processingthe data to put it into a more useful form or format. Loading refers tothe process of loading the data into the tables of a relationaldatabase.

Attempts have been made to use customer-focused NLP systems from thehospitality and restaurant industries in other spaces, but thesesystems' lack of specificity for particular uses make them inaccurateand ineffective for actionizing feedback. Further, investments in suchtechnologies do not yield the comprehensive, reliable, or actionableinformation necessary to improve an organization's viability, forexample, a healthcare organization. Instead, the value-added by the datareviewed by these technologies is diminished as true data integrationand interoperability is not achieved.

There have been few attempts to construct healthcare-specific oremployee engagement-specific NLP systems that may automatically collectand annotate key information related to the patient's care experienceand satisfaction, such as the patient's sentiment regarding theexperience, identification of key staff involved in the experience andkey themes describing the care experience.

Performing these annotations with a high degree of accuracy has provento be a difficult task due to the complex nature of language, the manyways that a care experience concept can be expressed, the inherentcomplexity of the subject matter, and the distributed and varied natureof the available data sources. As a result, NLP software tends to belarge, expensive and complex, difficult to develop and maintain, anddemands significant processing power, working memory, and time to run.Further, when attempting to process data from isolated sources indiffering formats, annotation accuracy is difficult to achieve. This isespecially true for unstructured data—annotations regarding sentiments,named entities, key themes and the like that may fall below atraditional threshold for statistical significance. Nevertheless,unstructured data may indicate real problems with care experiences thatare of value to healthcare administrators. Despite its value, it hastraditionally been difficult to process and understand.

Furthermore, some methods of data extraction are slow and ineffective.These systems, however, which use only a fraction of the data available,may reduce cost and improve outcomes. If systems and methods had thecapability of using the knowledge incorporated within unstructured datain an efficient manner to improve experience, the benefits would betremendous. By using this knowledge, a user's experience could beimproved and cost reduced through quality improvement, efficiency,comparative effectiveness, safety, and other analytics powered by thisdata.

Thus, there is a need in the field of processing experience data, andmore specifically in the field of processing disparate data sources suchas medical records, employee records, government surveys, employeesurveys, review sites and social media, for new and improved systems andmethods for processing data. In particular, systems and methods areneeded that are able to rapidly parse, combine, and interpret multiplestructured and unstructured data sources. Described herein are devices,systems, and methods that address the problems and meet the identifiedneeds described above.

From the foregoing it is believed that those skilled in the pertinentart will recognize the meritorious advancement of the present disclosureand will readily understand that while the present disclosure has beendescribed in association with one or more embodiments thereof, and otherembodiments illustrated in the accompanying drawings, numerous changesmodification and substitutions of equivalents may be made thereinwithout departing from the spirit and scope of the present disclosurewhich is intended to be unlimited by the foregoing except as may appearin the following appended claim. Therefore, the embodiments of thepresent disclosure in which an exclusive property or privilege isclaimed are defined in the following appended claims.

The invention claimed is:
 1. A method for processing and actionizingexperience data, the method comprising: receiving, by a server, aplurality of communications, wherein each of the plurality ofcommunications comprises comment data, wherein the comment datacomprises structured or unstructured data; parsing, by the server, thecomment data from each of the plurality of communications for individualphrases to generate a plurality of phrases in response to receiving theplurality of communications; predicting, by the server with a naturallanguage processing (NLP) engine, one or more annotations for each ofthe plurality of phrases based upon a machine learning score in responseto parsing the comment data, wherein each of the one or more annotationscomprises a sentiment of the corresponding phrase and a primary tagindicative of a subject matter of the corresponding phrase; determining,by the server, whether a certainty associated with the one or moreannotations is less than a predetermined threshold based on the machinelearning score; and predicting, by the server, the one or moreannotations for the plurality of phrases based upon a reference score inresponse to determining that the certainty is less than thepredetermined threshold.
 2. The method of claim 1, wherein predictingthe one or more annotations comprises classifying the plurality ofphrases to identify the primary tag based on a predetermined ontology.3. The method of claim 2, wherein the predetermined ontology comprises ahealthcare domain ontology.
 4. The method of claim 1, further comprisinggenerating, by the server, a dashboard web page for a user that includesthe one or more annotations in response to predicting the one or moreannotations.
 5. The method of claim 1, further comprising receiving, bythe server, the reference score from an individual review.
 6. The methodof claim 1, further comprising: storing, by the server, the comment dataof each of the plurality of communications at a relational database; andstoring, by the server, the one or more annotations at the relationaldatabase, wherein each record corresponds to an annotation, and whereineach record includes the sentiment and the primary tag.
 7. The method ofclaim 1, wherein predicting the one or more annotations comprisesidentifying a named entity in the plurality of phrases.
 8. The method ofclaim 1, wherein predicting the one or more annotations comprisesextracting a secondary tag indicative of a theme from the plurality ofphrases.
 9. A system for processing and actionizing experience data, thesystem comprising a server having a natural language processing (NLP)engine, wherein: the server is to: receive a plurality ofcommunications, wherein each of the plurality of communicationscomprises comment data, wherein the comment data comprises structured orunstructured data; and parse the comment data from each of the pluralityof communications for individual phrases to generate a plurality ofphrases in response to receiving the plurality of communications; theNLP engine is to predict one or more annotations for each of theplurality of phrases based upon a machine learning score in response toparsing the comment data, wherein each of the one or more annotationscomprises a sentiment of the corresponding phrase and a primary tagindicative of a subject matter of the corresponding phrase, and whereinthe server is further to determine whether a certainty associated withthe one or more annotations is less than a predetermined threshold basedon the machine learning score; and predict the one or more annotationsfor the plurality of phrases based upon a reference score in response toa determination that the certainty is less than the predeterminedthreshold.
 10. The system of claim 9, wherein to predict the one or moreannotations comprises to classify the plurality of phrases to identifythe primary tag based on a predetermined ontology.
 11. The system ofclaim 10, wherein the predetermined ontology comprises a healthcaredomain ontology.
 12. The system of claim 9, wherein the server isfurther to generate a dashboard web page for a user that includes theone or more annotations in response to prediction of the one or moreannotations.
 13. The system of claim 9, wherein the server is further toreceive the reference score from an individual review.
 14. The system ofclaim 9, further comprising a relational database, wherein the server isfurther to: store the comment data of each of the plurality ofcommunications at the relational database; and store the one or moreannotations at the relational database, wherein each record correspondsto an annotation, and wherein each record includes the sentiment and theprimary tag.
 15. The system of claim 9, wherein to predict the one ormore annotations comprises to identify a named entity in the pluralityof phrases.
 16. The system of claim 9, wherein to predict the one ormore annotations comprises to extract a secondary tag indicative of atheme from the plurality of phrases.
 17. One or more non-transitorycomputer-readable storage media comprising a plurality of instructionsstored thereon that, in response to being executed, cause a server to:receive a plurality of communications, wherein each of the plurality ofcommunications comprises comment data, wherein the comment datacomprises structured or unstructured data; parse the comment data fromeach of the plurality of communications for individual phrases togenerate a plurality of phrases in response to receiving the pluralityof communications; predict one or more annotations for each of theplurality of phrases based upon a machine learning score in response toparsing the comment data, wherein each of the one or more annotationscomprises a sentiment of the corresponding phrase and a primary tagindicative of a subject matter of the corresponding phrase; determinewhether a certainty associated with the one or more annotations is lessthan a predetermined threshold based on the machine learning score; andpredict the one or more annotations for the plurality of phrases basedupon a reference score in response to determining that the certainty isless than the predetermined threshold.
 18. The one or morenon-transitory computer-readable storage media of claim 17, wherein topredict the one or more annotations comprises to classify the pluralityof phrases to identify the primary tag based on a predeterminedontology.
 19. The one or more non-transitory computer-readable storagemedia of claim 18, wherein the predetermined ontology comprises ahealthcare domain ontology.
 20. The one or more non-transitorycomputer-readable storage media of claim 17, further comprising aplurality of instructions stored thereon that, in response to beingexecuted, cause the server to generate a dashboard web page for a userthat includes the one or more annotations in response to predicting theone or more annotations.
 21. The one or more non-transitorycomputer-readable storage media of claim 17, further comprising aplurality of instructions stored thereon that, in response to beingexecuted, cause the server to receive the reference score from anindividual review.
 22. The one or more non-transitory computer-readablestorage media of claim 17, further comprising a plurality ofinstructions stored thereon that, in response to being executed, causethe server to: store the comment data of each of the plurality ofcommunications at a relational database; and store the one or moreannotations at the relational database, wherein each record correspondsto an annotation, and wherein each record includes the sentiment and theprimary tag.
 23. The one or more non-transitory computer-readablestorage media of claim 17, wherein to predict the one or moreannotations comprises to identify a named entity in the plurality ofphrases.
 24. The one or more non-transitory computer-readable storagemedia of claim 17, wherein to predict the one or more annotationscomprises to extract a secondary tag indicative of a theme from theplurality of phrases.